[SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference #53718
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JIRA Issue Information=== Sub-task SPARK-54938 === This comment was automatically generated by GitHub Actions |
Member
zhengruifeng
reviewed
Jan 8, 2026
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py
Outdated
Show resolved
Hide resolved
zhengruifeng
reviewed
Jan 8, 2026
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py
Outdated
Show resolved
Hide resolved
zhengruifeng
reviewed
Jan 8, 2026
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py
Outdated
Show resolved
Hide resolved
zhengruifeng
reviewed
Jan 8, 2026
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py
Outdated
Show resolved
Hide resolved
zhengruifeng
reviewed
Jan 8, 2026
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py
Outdated
Show resolved
Hide resolved
zhengruifeng
reviewed
Jan 9, 2026
Contributor
zhengruifeng
left a comment
zhengruifeng
left a comment
There was a problem hiding this comment.
thanks so much, it is much cleaner.
inspiried by https://github.com/apache/spark/pull/53727/changes,
I think we need to also test following cases:
1, string: non-english values;
2, integral: min max values, and make it overflow
3, floats: nan, -inf, inf
4, time: Unix epoch, min max values
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py
Outdated
Show resolved
Hide resolved
python/pyspark/tests/upstream/pyarrow/test_pyarrow_type_inference.py
Outdated
Show resolved
Hide resolved
Yicong-Huang
force-pushed
the
SPARK-54938/test/add-tests-for-pa-array-type-inference
branch
from
efbc505 to
905b616
Compare
Yicong-Huang
force-pushed
the
SPARK-54938/test/add-tests-for-pa-array-type-inference
branch
from
905b616 to
f12b2a0
Compare
dev/sparktestsupport/modules.py
Outdated
| # unittests for upstream projects | ||
| "pyspark.tests.upstream.pyarrow.test_pyarrow_ignore_timezone", | ||
| "pyspark.tests.upstream.pyarrow.test_pyarrow_scalar_type_inference", | ||
| "pyspark.tests.upstream.pyarrow.test_pyarrow_type_inference", |
Contributor
There was a problem hiding this comment.
let's rename the file test_pyarrow_array_type_inference
zhengruifeng
approved these changes
Jan 13, 2026
Contributor
|
thanks, merged to master |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add tests for PyArrow's
pa.arraytype inference behavior. These tests monitor upstream PyArrow behavior to ensure PySpark's assumptions remain valid across versions.The tests cover type inference across input categories:
Nonevalueslist,tuple,dict(struct)Types tested include:
Pandas extension types tested:
pd.Int8Dtype()...pd.Int64Dtype(),pd.UInt8Dtype()...pd.UInt64Dtype(),pd.Float32Dtype(),pd.Float64Dtype(),pd.BooleanDtype(),pd.StringDtype()pd.ArrowDtype(pa.int64()),pd.ArrowDtype(pa.float64()),pd.ArrowDtype(pa.large_string()), etc.Why are the changes needed?
This is part of SPARK-54936 to monitor behavior changes from upstream dependencies. By testing PyArrow's type inference behavior, we can detect breaking changes when upgrading PyArrow versions.
Does this PR introduce any user-facing change?
No. This PR only adds tests.
How was this patch tested?
New unit tests added:
Was this patch authored or co-authored using generative AI tooling?
No.